INTEGRATED VISION SYSTEM 



PACKGROUNP O F THS INVENTION 

The present invention relates to an integrated vision 
system that provides crew on vehicle with views of high visibility 
even at actually low visibility. 

Vehicles , for example, aircraft are provided with a vision 
system having image sensors such as an infrared camera, a 
milli-wave radar and a laser radar. The vision system offers a 
driver or pilot with artificial pseudo -views generated based on 
view data collected by the image sensors at low visibility at 
night or in bad weather and three-dimensional (3-D) map data 
stored in the system for safety. 

Japanese -Unexamined Patent Publication No. 11-72350 
discloses generation of pseudo-views using wide-area 
geographical data based on 3-D map stored in memory and data on 
obstacles such as high-voltage electrical power lines, 
skyscrapers and cranes and displaying pseudo -views and actual 
views overlapping with each other on a transparent -type display 
mounted on a helmet of a pilot. 

View data collected by the image sensors such as an 
infrared camera , a milli-wave radar and a laser radar are , however , 
not sufficient for a driver or pilot. Moreover, 3-D map data 
cannot follow actual changing geographical conditions . 
Pseudo-views generated based on these data therefore do not meet 
the requirements of a driver or pilot . 

In detail, infrared cameras can be used at certain level 
of low visibility, particularly, can generate extremely clear 
images at night , however, lack in reality, perspective and feeling 
of speed due to monochrome images. 

Milli-wave radars can cover relatively long rage even 
in rainy weather, thus useful in image displaying at low 
visibility, however, cannot generate clear images due to 
wavelength extremely longer than light, thus not sufficient for 
a driver or pilot . 

Laser radars have an excellent obstacle detecting 
function, however, take long for scanning a wide area, thus 
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revealing low response. For a narrow scanning area, they provide 
relatively clear images, but, narrow views for a driver or pilot, 
thus not sufficient for safety. 

Generation of images of scenery in wide range of sight 
5 which could be viewed by a driver or pilot is useful. Such image 
generation requires decision of degree of risk of collision based 
on comparison among geographical data, obstacle data and vehicle 
positional data (longitude, latitude and altitude) . These data, 
however, may not match actual land features and obstacles. Such 
1 0 image generation thus has a difficulty in covering newly appearing 
obstacles and requires a lot of confirmation of safety. 



SUMMARY QF THE INVENTION 
A purpose of the present invention is to provide an 

1 5 integrated vision system that offers crew on a vehicle with almost 
real pseudo views at high visibility like in a good whether even 
at low visibility with detection of obstacles to the front for 
safe and sure flight or driving. 

The present invention provides an integrated vision system 

20 comprising: at least one stereo -camera installed in a vehicle 
for taking images of predetermined outside area; a stereo-image 
recognizer for processing a pair of images taken by the 
stereo-camera to recognize objects that are obstacles to the front , 
thus generating obstacle data; an integrated view data generator 

25 for generating integrated view data including three-dimensional 
view data based on the pair of images taken by the stereo -camera 
and the obstacle data from the stereo -image recognizer; and an 
integrated image display for displaying the integrated view data 
as visible images to crew on the vehicle. 
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BRIEF DESCRIPTION QF DRAWINGS 
FIG. 1 shows a block diagram of an integrated vision system 
according to the present invention; and 

FIG. 2 illustrates displaying zones. 



35 



DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
Preferred embodiments according to the present invention 



will be disclosed with reference to the attached drawings. 

An integrated vision system 1 shown in FIG. 1 is installed 
in a vehicle such as an automobile, a train, and an aircraft. 
The system 1 offers integrated views to a driver or pilot generated 
as visible images of virtual reality at high visibility like in 
good weather even though actual visibility is very low in bad 
weather due to mist or fog, or at night. 

Disclosed hereinafter is an embodiment in which the 
integrated vision system 1 is installed in an aircraft such as 
a helicopter that flies at relatively low altitude. 

The integrated vision system 1 is provided with a 
stereo-camera 2 for taking images of forward scenery of a 
predetermined area, an image combining apparatus 10 and an 
integrated view displaying apparatus 20 as main components. 

A pair of left and right images taken by the stereo -camera 
2 are displayed on left and right viewing points of a pilot to 
generate three-dimensional (3-D) images giving perspective and 
feeling of altitude and speed to the pilot . 

Moreover, the pair of left and right images are processed 
by stereo- image processing for calculation of data on (relative) 
distance to objects . The image and distance data are processed 
by image recognition processing for displaying obstacles as 
warning when the obstacles are detected on or in the vicinity 
of flight route. 

The integrated vision system 1 is provided with a 
sight -axis switch 3 for varying an axis of sighting by which the 
stereo-camera 2 turns into the direction required by the pilot 
or other crew, a display-mode switch 4 for controlling the stereo 
camera 2 to halt displaying 3-D images, and flight data interface 
5 for entering flight data such as speed, altitude, position and 
attitude of a helicopter. 

The sight -axis switch 3 is useful to know beforehand the 
conditions of flight route to which the helicopter is to turn 
into or determine whether there is any obstacle on the route. 
The switch 3 in this embodiment is a manual switch to manually 
rotate the optical axis of the stereo-camera 2. Not only this, 
the stereo-camera 3 can be automatically turned into any direction 
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by detecting pilot's viewing point by a head-motion tracker 23, 
etc., which will be described later. 

The stereo-camera 2 in this embodiment consists of two 
infrared cameras for generating extremely clear images 
5 particularly at night. The two infrared cameras are arranged 
with an optimum distance (base-line length) within an allowable 
range based on search range and distance accuracy for accurately 
detecting obstacles predicted under several flight conditions. 

Flight conditions under which a pilot requires support of 

10 artificial view at general low visibility mostly include night 
flight or other flight very close to this. Infrared cameras 
having an excellent night vision function are useful in such 
conditions. Particularly, 3-D images generated by two infrared 
cameras offer virtual reality to a pilot with perspective and 

15 feeling of altitude and speed which cannot be achieved by a single 
infrared camera. 

Other flight conditions can be covered by several types 
of image sensors such as an ordinary camera, an intensifier that 
responses faint light, active /passive milli-wave cameras 

20 exhibiting excellent transparency to mist and rain and sensitive 
CCDs , other than infrared cameras . These sensors can be 
selectively combined in accordance with cause of low visibility. 

When relatively light-weight image sensors are used for 
cameras of the stereo camera 2, the base-line length for the 

25 stereo- camera 2 can be set by shifting both or either of cameras 
with the image sensors. 

Heavy image sensors can be fixed with an objective lens 
mechanism installed in a tube of a periscope, the tube- length 
being variable for varying the base-line length. 

30 The image combining apparatus 10 is provided with a 

stereo -image recognition processor 11 for recognizing obstacles 
by processing left and right images from the stereo camera 2, 
a geographical image generator 12 for generating 3-D geographical 
images of scenery which could be viewed by a pilot or crew based 

35 on view-point data sent from the head-motion tracker 23 which 
will be described later and data from the flight data interface 
5, and an integrated view generator 13 for generating integrated 
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views which are combination of 3-D view data of left and right 
images from the stereo-camera 2, obstacle data from the 
stereo-image recognition processor 11, geographical data on wide 
view from the geographical image generator 12 and data from the 
5 flight data interface 5. 

The stereo-image recognition processor 11 is provided with 
an image database 11a that stores several types of 3-D data for 
recognizing and displaying several types of obstacles, etc. 

The geographical image generator 12 is provided with a 3-D 

10 digital map 12a that stores wide-area geographical data obtained 
by aerial survey or from satellites. 

The integrated view displaying apparatus 20 is provided 
with a head-mount display (HMD) 21 to be mounted on a helmet of 
the pilot or crew and having a transparent -type display such as 

1 5 a transparent -type liquid-crystal display panel by which a pilot 
or crew can view actual scenery through integrated views from 
the integrated view generator 13, a display adjuster 22 for 
adjusting intensity and contrast of integrated views, and 
transparency to the actual views on the HMD 21 so that the pilot 

20 or crew can observe the overlapped actual views and integrated 
views in a good condition, and the head-motion tracker 23 for 
tracking the head position and attitude of the pilot or crew to 
output view-point data of the pilot or crew. 

A pair of left and right images taken by the stereo-camera 

25 2 during flight are sent to the integrated view generator 13 as 
3-D images and also to the stereo-image recognition processor 
11 for detecting forward obstacles. 

The stereo-image recognition processor 11 processes the 
left and right images from the stereo-camera 2 by stereo -matching 

30 processing to obtain correlation between the images, thus 
calculating distance data by triangular surveying based on 
parallax to the same object, the position of the stereo camera 
2 and its parameter such as focal length. 

Data stored in the image database is accessed based on the 

35 distance and image data for recognizing obstacles, any objects 
that would block flight. The integrated vision system 1 
installed in aircraft, for example, helicopters that fly 
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relatively low altitude recognizes structures such as pylons and 
skyscrapers or other aircraft , etc. , in forward view during night 
flight. When the system 1 also recognizes high-voltage 
electrical power lines in the image data under recognition of 
5 pylons, it generates obstacle data that symbolizes or emphasizes 
the power lines and send the data to the integrated view generator 
13. 

Accordingly, when the integrated vision system 1 
recognizes pylons, high-voltage electrical power lines and 

10 skyscraper, etc., in forward view via the stereo camera 2, it 
immediately warns the pilot of those structures as obstacles that 
could collide with so that the helicopter immediately takes an 
evasive action. 

The present invention therefore offers high safety flight 

15 or driving without determination of degree of risk of collision 
by comparing stored positional data such as longitude, latitude 
and altitude with actual flight data in 3-D digital map. 

The geographical image generator 12 performs 
coordinate-conversion of aircraft data such as speed, altitude 

20 and attitude and flight positional data input via the aircraft 
flight data interface 5 onto viewing points of the pilot or crew 
input from the head-motion tracker 23. The generator 12 further 
retrieves the converted data from the 3-D digital map 12a as 3-D 
geographical images which could be viewed by the pilot or crew 

25 and sent the 3-D Images to the integrated view generator 13. The 
3-D images are geographical data wider than actual scenery in 
forward view, for example, a row of mountains in the distance 
which will not be directly connected to safety against collision. 
These 3-D geographical image data can been seen as almost real 

30 scenes based on 3-D display generated by computer graphics with 
geographical information such as place names, lakes, loads and 
rivers if necessary. 

The 3-D geographical image data may be generated from 
images detected by a wide milli-wave radar instead of the 3-D 

35 digital map 12a. 

The integrated view generator 13 receives 3-D image data 
of left and right images taken by the stereo-camera 2 controlled 
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by the sight -axis switch 3 and also the obstacle images generated 
by the stereo-image recognition processor 11. The generator 13 
also receives wide peripheral geographical images generated by 
the geographical image generator 12. 
5 The integrated view generator 13 combines the obstacle and 

peripheral geographical images with image processing such as 
adjustments to resolution, intensity, contrast and color, and 
also edge-blending under the control of the display-mode switch 
4, to generate natural images with no visible joints of 
10 combination as integrated view data. 

The integrated view generator 13 combines the integrated 
view data with flight data such as speed, altitude, position and 

0 attitude sent from the flight data interface 5 if necessary, and 
^ send them to the HMD 21. 

01 15 At high visibility such as a good weather , the stereo camera 
~f\ 2 may be turned off so that integrated view data not 3-D images 
%J of the stereo camera 2 but the obstacle data and other data, if 
^ necessary, are sent to the HMD 21. 

q FIG. 2 illustrates viewing zones covered by the HMD 21 with 

*H 20 the integrated view data processed as disclosed above. 

n A viewing zone surrounded by a dashed line is used for 

S displaying the forward view data with obstacle data from the 

stereo camera 2 . Another viewing zone surrounded by a dotted line 
but outside the dashed line is used for displaying the wide- 
25 area view data of 3-D geographical images. 

The pilot or crew can see actual scenery from a cockpit 
30 through windows overlapping with the integrated views while 
watching indicators on the cockpit, thus the pilot or crew can 
see to indicated data in addition to those displayed on the HMD 
30 21. 

As disclosed above, the integrated vision system 1 
according to the present invention offers a pilot or crew with 
3-D images almost real in perspective, altitude and speed based 
on left and right images taken by the stereo camera 2 even at 
35 a low visibility, which cannot be achieved by a single camera. 

The integrated vision system 1 further processes the left 
and right images from the stereo camera 2 by stere-image 
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processing for image recognition using distance data to detect 
obstacles on or in the vicinity of flight rout and displays the 
obstacles as warning. 

The integrated vision system 1 according to the present 
5 invention thus offers pseudo-visual flight even at row visibility 
to support a pilot for safe and sure flight in regular service 
or emergency. 

Three -D images may not be required at high visibility, 
however, detection of obstacles for warning by stereo -image 

10 recognition processing achieves further safe flight. 

Three -D image display of forward views and obstacle 
detection/warning are performed by the stereo-camera 2 with no 
sensors for respective functions. The integrated vision system 
1 according to present invention thus can be structured as a 

15 reliable and light system at a low cost. 

As disclosed above, the present invention offers crew on 
a vehicle with almost real pseudo views at high visibility like 
in a good whether even at low visibility for safe and sure flight 
or driving with detection of obstacles to the front. 

20 It is further understood by those skilled in the art that 

the foregoing description is a preferred embodiment of the 
disclosed system and that various change and modification may 
be made in the invention without departing from the spirit and 
scope thereof. 



